8 research outputs found
Clustering the Sketch: A Novel Approach to Embedding Table Compression
Embedding tables are used by machine learning systems to work with
categorical features. In modern Recommendation Systems, these tables can be
very large, necessitating the development of new methods for fitting them in
memory, even during training. We suggest Clustered Compositional Embeddings
(CCE) which combines clustering-based compression like quantization to
codebooks with dynamic methods like The Hashing Trick and Compositional
Embeddings (Shi et al., 2020). Experimentally CCE achieves the best of both
worlds: The high compression rate of codebook-based quantization, but
*dynamically* like hashing-based methods, so it can be used during training.
Theoretically, we prove that CCE is guaranteed to converge to the optimal
codebook and give a tight bound for the number of iterations required
Parameter-free Locality Sensitive Hashing for Spherical Range Reporting
We present a data structure for *spherical range reporting* on a point set
, i.e., reporting all points in that lie within radius of a given
query point . Our solution builds upon the Locality-Sensitive Hashing (LSH)
framework of Indyk and Motwani, which represents the asymptotically best
solutions to near neighbor problems in high dimensions. While traditional LSH
data structures have several parameters whose optimal values depend on the
distance distribution from to the points of , our data structure is
parameter-free, except for the space usage, which is configurable by the user.
Nevertheless, its expected query time basically matches that of an LSH data
structure whose parameters have been *optimally chosen for the data and query*
in question under the given space constraints. In particular, our data
structure provides a smooth trade-off between hard queries (typically addressed
by standard LSH) and easy queries such as those where the number of points to
report is a constant fraction of , or where almost all points in are far
away from the query point. In contrast, known data structures fix LSH
parameters based on certain parameters of the input alone.
The algorithm has expected query time bounded by , where
is the number of points to report and depends on the data
distribution and the strength of the LSH family used. We further present a
parameter-free way of using multi-probing, for LSH families that support it,
and show that for many such families this approach allows us to get expected
query time close to , which is the best we can hope to achieve
using LSH. The previously best running time in high dimensions was . For many data distributions where the intrinsic dimensionality of the
point set close to is low, we can give improved upper bounds on the
expected query time.Comment: 21 pages, 5 figures, due to the limitation "The abstract field cannot
be longer than 1,920 characters", the abstract appearing here is slightly
shorter than that in the PDF fil
Oblivious Sketching of High-Degree Polynomial Kernels
Kernel methods are fundamental tools in machine learning that allow detection
of non-linear dependencies between data without explicitly constructing feature
vectors in high dimensional spaces. A major disadvantage of kernel methods is
their poor scalability: primitives such as kernel PCA or kernel ridge
regression generally take prohibitively large quadratic space and (at least)
quadratic time, as kernel matrices are usually dense. Some methods for speeding
up kernel linear algebra are known, but they all invariably take time
exponential in either the dimension of the input point set (e.g., fast
multipole methods suffer from the curse of dimensionality) or in the degree of
the kernel function.
Oblivious sketching has emerged as a powerful approach to speeding up
numerical linear algebra over the past decade, but our understanding of
oblivious sketching solutions for kernel matrices has remained quite limited,
suffering from the aforementioned exponential dependence on input parameters.
Our main contribution is a general method for applying sketching solutions
developed in numerical linear algebra over the past decade to a tensoring of
data points without forming the tensoring explicitly. This leads to the first
oblivious sketch for the polynomial kernel with a target dimension that is only
polynomially dependent on the degree of the kernel function, as well as the
first oblivious sketch for the Gaussian kernel on bounded datasets that does
not suffer from an exponential dependence on the dimensionality of input data
points
Hardness of Approximate Nearest Neighbor Search
We prove conditional near-quadratic running time lower bounds for approximate
Bichromatic Closest Pair with Euclidean, Manhattan, Hamming, or edit distance.
Specifically, unless the Strong Exponential Time Hypothesis (SETH) is false,
for every there exists a constant such that computing a
-approximation to the Bichromatic Closest Pair requires
time. In particular, this implies a near-linear query time for
Approximate Nearest Neighbor search with polynomial preprocessing time.
Our reduction uses the Distributed PCP framework of [ARW'17], but obtains
improved efficiency using Algebraic Geometry (AG) codes. Efficient PCPs from AG
codes have been constructed in other settings before [BKKMS'16, BCGRS'17], but
our construction is the first to yield new hardness results